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Title:  Automated  analysis  and  Display  of  Temporal  Sequences  of  Mammograms 

Walter  F.  Good,  Ph.D. 


Introduction 

Considerable  attention  has  been  paid  to  methods  for  improving  the  diagnostic  accuracy  and 
efficiency  of  mammographic  screening  procedures,  including:  Assessments  of  the  importance  of  age  of 
patients  to  be  screened  [1];  Appropriate  screening  intervals  [2];  Appropriate  screening  techniques  [3,4]; 
Recommended  number  of  views  per  examination  [5];  Single  vs.  double  reading  [6];  and,  Use  of  prior 
examinations  for  comparison  [7]. 

Customarily,  in  screening  programs  mammographers  only  compare  the  current  exam  to  one 
previous  exam  (e.g.,  one  year  prior  or  two  years  prior).  This  is  partly  for  efficiency  but  also  because  there  is 
a  lack  of  compelling  evidence  on  how  changing  tissue  patterns  relate  to  the  likelihood  of  developing  a 
malignancy.  Retrospective  inspection  of  exams  taken  in  years  prior  to  the  detection  of  a  malignancy  often 
indicates  the  presence  of  an  abnormality.  Such  anecdotal  evidence  suggests  that  it  is  at  least  possible  that 
series  of  screening  exams  contain  additional  information  that  could  improve  the  accuracy  of  the  diagnostic 
process. 

There  is  also  evidence  that  difference  images  in  certain  procedures  can  contain  diagnostically  useful 
information,  despite  the  presence  of  substantial  subtraction  artifact.  In  the  diagnosis  of  chest  radiography,  it 
has  been  demonstrated  that  adding  temporally  subtracted  images  to  the  diagnostic  process  can,  in  many 
situations,  significantly  improve  diagnostic  accuracy  and  reduce  mean  interpretation  time  [8],  Difference 
images  obtained  by  temporal  subtraction  methods  in  mammography  have  properties  somewhat  similar  to 
images  obtained  with  bilateral  subtraction  images  used  in  CAD,  where  it  can  be  shown  that  differences 
contain  diagnostic  information  [9-13].  However,  like  the  subtraction  of  chest  images,  temporal  image 
subtraction  of  mammograms  is  a  difficult  task  due  to  tissue  changes  over  time,  inconsistencies  in  breast 
compression,  skewing  of  the  three-dimensional  breast  relative  to  the  image  projection  during  compression 
and  lack  of  easy  landmarks  on  the  breast  that  can  be  used  to  facilitate  optimal  image  registration.  Previous 
attempts  to  develop  acceptable  subtraction  methods  for  mammograms,  including  methods  for  both  bilateral 
subtraction  [9-12]  and  temporal  subtraction  [14-16],  have  had  only  partial  success.  As  a  result,  the  temporal 
subtraction  of  mammograms  has  not  been  employed  clinically. 

In  an  attempt  to  improve  diagnostic  accuracy  of  screening  mammography,  this  project  is  adapting 
and  integrating  several  novel  technologies,  under  development  in  our  laboratory,  into  a  system  for 
providing  mammographers  with  information  about  changing  tissue  patterns,  and  the  corresponding 
likelihood  of  malignancy,  derived  from  temporal  sequences  of  images.  Specifically,  these  technologies 
include:  1)  A  method  for  locally  deforming  and  registering  mammograms  [17,18];  2)  Multi-image  CAD, 
which  we  have  previously  used  to  identify  corresponding  features  in  ipsilateral  views  [19];  and,  3) 
Adjustment  of  digitized  mammograms  to  account  nonuniform  tissue  thickness  due  to  breast  compression, 
to  enhance  soft  display  of  mammograms  and  to  enable  more  accurate  computerized  estimation  of  tissue 
composition  [20],  Our  hypotheses  are:  1)  Sequences  of  screening  mammograms  contain  information  about 
tissue  changes  that  is  not  otherwise  being  exploited  in  the  diagnosis  of  breast  cancer;  2)  Changing  tissue 
patterns  can  automatically  be  identified,  and  correlated  with  diagnostic  questions;  and,  3)  Such  information 
can  be  made  available  to  mammographers,  in  an  efficient  manner,  through  displays  specifically  designed 
for  these  image  sequences. 
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The  main  objectives  of  this  project  are  to  provide  a  system,  which  can  be  employed  at  the  discretion 
of  mammographers,  to:  1)  Normalize  images  to  facilitate  comparisons;  2)  Apply  our  previously  developed 
multi-image  CAD  methods  to  identify  corresponding  features  between  images,  and  evaluate  possible  trends 
for  any  such  features;  3)  Detect  and  classify  trends  in  temporal  sequences  (e.g.,  distinguish  between  normal 
and  abnormal  age-dependent  changes,  and  identify  those  patterns  of  change  that  are  associated  with  the 
development  of  malignancies);  4)  Calculate  and  present  various  kinds  of  parameter  images;  and,  5) 
Develop  a  display  system  for  efficiently  presenting  sequences  of  exams  to  mammographers,  with  the 
optional  ability  to  emphasize  trends  by  correcting  images  for  differences  in  exposure,  compression,  and 
local  misregistration.  Our  expectation  is  that  this  display  will  improve  the  diagnostic  performance  of 
mammographers  for  these  cases. 


Body 

Progress  on  this  project  closely  followed  the  schedule  and  methods  outlined  in  the  original 
proposal.  All  tasks  described  in  the  statement  of  work  have  been  undertaken  and  completed  successfully. 
The  tasks  that  were  completed  during  the  past  year  include: 

Task  4\  Image  analysis  software  development  (Months  6  -  30). 
k.  Detection  and  classification  of  trends. 

Task  7:  Final  training  and  testing  of  classifier  (Months  24-30). 

Task  8:  Final  processing  of  all  cases  (Months  30  -  34). 

Task  9:  Data  analysis  and  final  report  (Months  30-36). 

Throughout  the  project  certain  technical  issues  arose  and  these  are  being  addressed  in  the  one- 
year  extension  that  has  been  granted.  The  following  paragraphs  summarize  the  work  that  has  been 
completed  to  date. 

Task  1:  Select,  verify  and  digitize  cases  (Months  1-30) 

a.  A  set  of  240  sequences  of  screening  mammograms  will  be  selected  and  verified 

b.  All  verified  cases  will  be  digitized 

A  total  of  240  cases,  have  been  collected  since  this  project  was  funded.  Some  of  these  were 
collected  from  other  institutions.  Of  these,  125  are  negative  and  115  contain  malignancies.  All  cases  have 
been  verified,  digitized,  and  assembled  into  a  database.  The  database  has  been  anonymized  by  use  of  an 
honest  broker  system,  to  comply  with  the  requirements  of  HIPPA.  This  completes  the  active  acquisition 
of  cases  within  this  project. 

Case  selection  was  based  on  a  historical  prospective  approach,  in  which  we  examined  cases  that 
had  at  least  two  prior  screening  examinations.  We  only  considered  cases  for  which,  prior  to  the  latest 
examination  in  which  a  positive  abnormality  may  have  been  detected,  all  the  mammograms  for  these 
patients  had  been  interpreted  as  negative.  Each  case  consists  of  two  views  of  a  single  breast.  Originally, 
we  had  expected  to  select  all  cases  from  routine  mammographic  examinations  performed  at  the  UPMC 
and  Magee  health  care  systems,  but  to  find  a  sufficient  quantity  of  positive  cases  having  the  appropriate 
historical  sequence,  we  have  included  images  from  other  institutions.  Positive  cases  were  drawn  from 
cases  that  have  undergone  biopsy.  Negative  cases,  for  the  most  part,  are  cases  that  have  not  been 
recommended  for  biopsy,  but  in  some  instances,  subtle  negative  cases,  which  have  had  biopsies  that  were 
negative  for  the  presence  of  a  mass  or  microcalcification  cluster,  have  been  selected. 
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Each  selected  film  has  been  digitized  with  a  12  bit  modified  film  digitizer  (Lumisys)  having  an 
MTF  of  27  and  24%  at  the  Nyquist  frequency  (10  lp/mm)  in  the  X  and  Y  directions,  respectively.  The 
digitizer  is  routinely  evaluated  and  adjusted  to  maintain  spot  size,  linearity,  geometric  fidelity,  stability 
and  acceptable  spatial  noise  characteristics,  and  to  maintain  a  linear  relationship  between  pixel  value  and 
optical  density,  as  part  of  our  quality  assurance  procedures. 

Task  2:  Modification  of  workstation  for  this  project  (Months  1  -  12) 

An  existing  4-monitor  display,  which  was  developed  in  our  laboratory,  has  been  adapted 
for  this  study.  The  individual  monitors  (Clinton  Electronics,  Model  DS5000)  are  21  inch  portrait 
monitors,  having  a  resolution  of  2048  x  2560  pixels,  150  ftL  light  output  and  P45  phosphor.  The  displays 
are  driven  by  a  display  controller  (Dome  Imaging  Systems,  Model  Md5/PCX-2)  specifically  designed  for 
these  monitors,  and  calibrated  with  a  Barten  lookup  table  to  accommodate  the  contrast  sensitivity  of  the 
human  eye.  The  system  has  been  designed  to  display  four  images  of  a  single  exam  at  a  time,  or  a  single 
image  and  its  associated  parameter  images,  or  to  display  sequences  of  exams  sequentially,  at  a  rate 
determined  by  viewers.  This  task  was  completed  in  year  one. 

Task  3 :  Develop  display  software  (Months  3  -  15) 

a.  Display  difference  images 

b.  Display  trends 

The  ability  to  deform  sequentially  acquired  mammograms  so  that  they  can  be  placed  in  reasonable 
registration  is  a  central  aspect  of  this  project.  While  image  subtraction  was  addressed  somewhat  in  year 
one,  we  continued  the  development  of  the  image  deformation  and  subtraction  software  in  the  second  year. 
In  the  originally  proposed  method,  which  is  described  in  more  detail  under  section  g  of  Task  4  below,  we 
used  intersections  of  nipple  axis  lines  with  chest  walls  as  origins  of  polar  coordinate  systems  for 
performing  mappings  between  mammograms.  We  have  demonstrated  that  a  modification  of  this  approach 
is  preferable.  Specifically,  the  origin  is  now  defined  as  the  midpoint  of  the  chest  wall.  The  line  from  this 
point,  passing  through  the  nipple  serves  as  the  base  for  measuring  angles.  But  because  this  line  is  not 
generally  perpendicular  to  the  chest  wall,  a  nonlinear  monotonic  function  is  used  to  find  corresponding 
angles  between  images.  Viewers,  at  their  discretion,  have  the  option  of  either  subtracting  consecutive 
images  in  sequence  or  subtracting  each  previous  image  from  a  specific  (e.g.,  the  most  recent)  image.  Prior 
to  actually  performing  the  subtraction,  images  are  first  normalized  to  correct  for  tissue  thickness,  and  then 
geometrically  deformed  to  match  a  common  template.  The  details  of  this  procedure,  and  a  comparison  of 
the  various  techniques,  have  been  submitted  for  presentation  at  SPIE  Medical  Imaging  2005. 

Because  the  parameter  images  are  at  a  much  lower  effective  resolution  that  the  original 
mammographic  data,  these  can  be  displayed  side-by-side.  The  display  software  has  been  appropriately 
modified  to  allow  for  alternative  display  formats.  A  cine  mode  has  also  been  implemented  on  the  display 
system  as  one  means  of  displaying  temporal  sequences  of  image  data.  This  can  be  used  to  display  either 
the  original  mammographic  data  or  the  parameter  images  being  generated  from  each  mammographic 
image.  The  optimal  arrangement  of  the  displayed  information  will  depend  to  a  great  extent  on  the 
importance  of  the  various  kinds  of  parameter  and  trend  summary  images  being  generated. 

Task  4:  Image  analysis  software  development  (Months  6  -  30) 

a.  Correct  for  characteristic  curve 

b.  Correct  for  breast  compression 

c.  Skin  line  detection 

d.  Breast  compression  adjustment 

e.  Local  and  global  composition 
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f.  Global  composition  from  2  views 

g.  Local  geometric  registration 

h.  Histogram  equalization  between  images 

i.  Modify  Multi-View  CAD  for  this  project 

j.  Create  parameter  images 

k.  Detect  and  classify  trends 

Subtasks  a-e  and  g  were  largely  completed  during  year  one. 

a.  Correct  for  characteristic  curve  -  The  characteristic  curve  for  each  particular  film  was  obtained 
from  the  manufacturer.  Although  these  curves  are  generic,  they  are  sufficiently  accurate  for  this 
application,  given  other  imperfections  in  the  overall  imaging  chain.  Our  routine  quality  assurance  process 
on  the  digitizer  maintains  a  linear  relationship  between  pixel  value  and  optical  density.  The  characteristic 
curve  correction  algorithm  is  written  to  use  the  inverse  of  the  digitizer  characteristic  curve  to  convert  pixel 
values  to  optical  density  values,  and  then  us  the  inverse  of  the  film's  characteristic  curve  to  convert  optical 
density  to  film  exposure. 

b. d.  Correct  for  breast  compression  -  The  wide  dynamic  range  of  mammograms  is  caused  in  part 
by  the  non-uniformity  of  breast  thickness  during  breast  compression.  To  correct  for  this,  at  each  pixel  in  a 
mammogram,  we  determine  the  relative  thickness  of  the  compressed  breast  as  a  function  of  the  distance 
of  the  pixel  from  the  skin  line.  Although  there  are  computationally  more  efficient  methods,  we  measure 
the  distance  of  each  pixel  to  the  skin  line  with  a  simple  exhaustive  search.  We  divide  the  range  of  possible 
distances  into  a  small  number  of  intervals  (typically  32  to  64  depending  on  image  size)  and  for  each 
interval,  all  of  the  tissue  pixels  whose  distance  falls  within  that  interval  are  grouped.  For  each  such  group 
the  mean  and  standard  deviation  are  calculated,  and  an  appropriate  function  is  fitted,  with  constraints,  to 
the  means  plus  one  standard  deviation.  This  "correction"  function  represents  the  change  in  pixel  value 
with  respect  to  distance  from  the  skin  line,  and  actually  indicates  tissue  thickness  relative  to  this  distance. 
For  each  pixel,  the  correction  function  is  used  to  calculate  a  correction  value  from  the  pixel's  distance 
value,  and  this  correction  value  is  used  to  normalize  the  pixel  value.  In  the  central  regions  of  the  breast 
area,  the  correction  values  are  1  and  these  pixels  are  left  unchanged. 

c.  Skin  line  detection  -  In  this  project  it  is  necessary  to  make  geometric  measurements  that  are 
referenced  to  the  skin  line  and  nipple.  We  first  produce  a  low-resolution  version  of  the  image  by 
averaging  pixel  values  over  a  16016  block  and  then  apply  the  Sobel  gradient  to  the  reduced-resolution 
image.  At  each  point  in  the  image,  the  original  pixel  value,  the  magnitude  of  the  gradient,  the  direction  of 
the  gradient,  the  radial  distance  of  the  pixel  from  the  center  of  the  left  edge  of  the  film  and  a  direction 
value  for  the  surrounding  neighborhood  are  all  combined  to  produce  a  value  which  is  proportional  to  the 
likelihood  that  the  pixel  falls  on  the  skin  line.  These  likelihood  values  are  thresholded  and  then  placed  in 
an  image  where  they  are  processed  with  morphological  operators  to  eliminate  isolated  points.  The  points 
that  remain,  which  are  invariably  on  the  skin  line,  are  mapped  back  to  the  high-resolution  image  and  used 
to  fit  a  Bezier  Curve  to  define  a  smooth  skin  line.  Although  failure  is  rather  uncommon,  it  can  happen 
when  the  tissue  thickness  adjustment  process  fails  to  correctly  identify  the  skin  line.  This  can  occur  for 
images  where  the  skin  line  on  the  film  was  so  dense  (i.e.,  over  an  optical  density  of  about  4.2)  that  the 
digitizer  was  unable  to  distinguish  between  the  skin  line  and  the  image  background.  In  this  event,  we  do 
not  include  the  case  in  this  study. 

Detection  of  pectoral  muscle  -  To  detect  the  pectoral  muscle,  we  employ  a  template  matching 
with  the  mammographic  image,  which  is  a  standard  approach,  and  effective  in  cases  such  as  this  where 
structures  are  relatively  well  defined.  The  chest  wall  is  not  normally  visible  in  the  CC  view  so  we 
assume,  in  these  cases,  that  it  is  parallel  to,  and  0.5  cm  beyond  the  edge  of  the  image. 
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e.  Local  and  global  composition  -  After  correcting  images  for  characteristic  curves  of  film  and 
then  correcting  them  for  nonuniformities  due  to  breast  compression,  at  each  point  in  a  view,  we  estimate 
%  fibroglandular  tissue,  by  deriving  certain  features  directly  from  a  local  histogram  and  using  a  neural 
network,  trained  with  values  from  breast  MRI,  to  obtain  a  local  estimate  of  composition.  An  initial  global 
value  is  determined  for  the  image  by  integrating  over  the  tissue  area.  Tissue  composition  is  calculated 
independently  from  CC  and  MLO  views,  and  averaged  to  obtain  the  global  value  for  the  breast.  The  local 
values  in  each  of  the  two  views  are  then  scaled  appropriately. 

f.  Global  composition  from  two  views  -  As  implemented,  calculating  a  global  composition  for  a 
breast  from  two  views  simply  involves  averaging  the  values  from  the  CC  and  MLO  views  and  then 
adjusting  the  local  values  in  each  of  the  individual  views  so  that  they  match  the  averaged  value.  Values 
for  individual  views  are  found  by  the  following  procedure:  1)  Correct  images  for  characteristic  curve  of 
film;  2)  Detect  skin  line;  3)  Detect  pectoral  muscle;  4)  Correct  for  breast  compression;  5)  Calculate 
fraction  of  fibroglandular  tissue  at  each  pixel;  6)  Multiply  this  fraction  by  the  breast  thickness  at  the  pixel 
to  obtain  absolute  amount  of  fibroglandular  tissue  at  pixel;  and,  7)  Integrate  this  over  the  projected  area  of 
the  breast. 

g.  Local  geometric  registration  -  To  make  comparisons  between  sequentially  acquired  images 
more  feasible,  we  adjust  the  geometry  of  all  images  being  compared,  so  that  the  images  can  be  placed  in 
accurate  registration,  and  we  normalize  grayscales  so  that  they  reflect  equivalent  exposure  conditions.  In 
brief,  the  subtraction  method  uses  a  fully  automatic  nonlinear  transformation  that  can  map  any 
mammographic  image  onto  a  template  or  reference  image  while  assuring  concurrent  and  accurate 
registration  of  skin  lines,  nipples,  pectoral  muscles  and  nipple  axis  lines. 

The  geometric  transformation  is  diagramed  in  Figure  1 .  The  technique  begins  by  automatically 
detecting  pectoral  muscles  on  MLO  views  (e.g.,  OL  and  OX’  in  Figure  1),  skin  lines  and  nipple  locations, 
N  and  TV’,  in  Figure  1.  Polar  coordinate  systems  are  established  with  the  origins,  O  and  O’,  at  the 
intersection  of  the  nipple  axes  lines  (NALs),  ON  and  O  TV’,  and  lines  indicating  the  pectoral  muscles,  on 
MLO  views  or  with  lines  parallel  to,  and  0.5  cm  beyond,  the  edge  of  the  film  on  CC  views.  Tissue  pixels 

within  a  mammogram  are  identified  by  a 
relative  polar  coordinate,  which  we  define  to  be 
the  angle,  a,  of  their  position  vector  relative  to 
the  NAL  and  their  fractional  distance  between 
the  origin  and  the  skin  line.  For  each  pixel  P  in 
a  reference  template  (e.g.,  left  image  in  Figure 
1),  a  point  P’,  having  the  same  relative  polar 
coordinate,  is  found  in  the  image  to  be 
deformed.  The  pixel  at  location  P  in  the 

template  is  then  given  the  value  of  the  point  at 
P’,  as  determined  by  the  nearest  pixel  or 
through  interpolation,  after  adjusting  for  the 
local  Jacobian  of  the  transformation  at  the 
particular  point.  Clearly,  this  transformation 
maps  nipples  onto  nipples,  skin  pixels  onto  skin 
pixels  and  chest  wall  pixels  onto  chest  wall  pixels,  while  interior  pixels  are  deformed  correspondingly. 

Grayscale  adjustment  -  Once  images  have  been  deformed  so  that  a  reasonable  correlation  between 
pixels  can  be  assumed,  grayscales  are  corrected  by  applying  a  standard  linear  or  nonlinear  regression 
(e.g.,  using  low  degree  polynomials)  to  pixel  value  pairs  for  the  corresponding  image  points.  Pixel  values 
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of  the  deformed  image  are  then  adjusted  by  the  regression  equation,  to  minimize  the  sum-of-squares 
difference,  with  respect  to  admissible  grayscale  changes. 

As  stated  above,  the  geometric  registration  method  described  above  has  been  modified  somewhat  to 
improve  performance.  In  the  originally  proposed  method,  we  used  the  intersection  of  the  nipple  axis  line 
with  the  chest  wall  as  the  origin  of  a  polar  coordinate  system  for  performing  a  mapping  between 
mammograms.  We  now  have  demonstrated  that  a  modification  of  this  approach  is  preferable.  Specifically, 
the  origin  is  now  defined  as  the  midpoint  of  the  chest  wall.  The  line  from  this  point,  passing  through  the 
nipple  now  serves  as  the  base  for  measuring  angles.  But  because  this  line  is  not  generally  perpendicular  to 
the  chest  wall,  a  nonlinear  monotonic  function  is  used  to  find  corresponding  angles  between  images. 

h.  Histogram  equalization  between  images  -  Once  images  have  been  deformed  so  that  a  reasonable 
correlation  between  pixels  can  be  assumed,  grayscales  are  corrected  by  fitting  all  pixel  value  pairs  for 
corresponding  image  points  with  a  3-segment  spline  function  consisting  of  two  quadratic  polynomials 
smoothly  joined  by  a  linear  segment.  Pixel  values  of  the  deformed  image  are  then  adjusted  by  the 
regression  equation,  to  minimize  the  sum-of-squares  difference,  with  respect  to  admissible  grayscale 
changes. 

i.  Modify  multi-view  CAD  for  this  project  -  Our  multi-view  CAD  program,  much  of  which  is  being 
adapted  for  use  in  this  project,  contains  methods  for  comparing  features  taken  at  the  same  time  from 
multiple  views.  For  this  project,  these  methods  have  been  modified  to  permit  their  use  on  images  taken  at 
different  times.  Specifically,  we  included  methods  for  applying  linear  regression  to  measure  changes  in 
features  with  respect  to  time.  The  main  difficulty  we  have  encountered  in  doing  this  is  that  our  ability  to 
find  corresponding  features  in  a  sequence  of  images  depends  on  our  ability  to  deform  the  images  so  that 
they  can  be  put  in  geometric  registration. 

i.  Create  parameter  images  -  A  general  method  for  creating  parameter  images  has  been  implemented. 
For  temporal  sequences  of  images,  all  images  are  deformed  to  the  shape  of  the  most  recent  image.  A  local 
feature  filter  is  applied  to  each  image  to  produce  a  corresponding  image  in  which  pixel  values  reflect  the 
local  magnitude  of  the  particular  feature  in  the  original  image. 

Local  features  -  We  have  defined  a  number  of  local  features  that  could  potentially  be  of  value  in 
identifying  changing  patterns  in  temporal  sequences  of  mammograms.  These  include:  local  breast  density, 
local  variance,  and,  various  measures  of  image  texture. 


k.  Detect  and  classify  trends  -  For  each  sequence  of  parameter  files  as  described  above,  we  have 
attempted  to  identify  both  local  and  global  trends.  Direct  information  about  local  trends,  such  as  correlation 
coefficients,  is  displayable.  To  create  the  trend  summary  image  for  a  feature,  at  each  pixel  we  calculate  a 
correlation  coefficient  and  the  slope  of  the  best-fit  line  to  the  sequence  of  parameter  values.  In  summary 
images  we  are  currently  using  hue  to  represent  slope  and  brightness  to  represent  correlation  (with  black 
corresponding  to  r2  =  0). 

As  a  prototype  for  developing  these  methods  we  used  the  local  breast  density  filter  that  produces 
images  somewhat  similar  in  appearance  to  the  original  mammograms,  but  at  a  lower  resolution.  In 
summary  images,  locations  are  assigned  values  from  a  scale  that  ranges  from  cyan  (falling  density)  to  red 
(increasing  density),  where  brightness  indicates  the  degree  of  correlation  (uncorrelated  sequences  of  pixel 
values  are  shown  in  black).  Other  filters  (e.g.,  local  variance,  local  fractal  dimension)  have  also  been 
implemented  and  behave  in  a  similar  manner.  The  results  of  these  methods  were  presented  at  the  Era  of 
Hope  Meeting  in  2005. 
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Task  5:  Initial  training  and  testing  of  classifier  (Months  16-18) 

In  addition  to  creating  parameter  images  corresponding  to  individual  features,  we  have  trained  a 
classifier  to  predict  the  likelihood  of  developing  a  malignancy  based  on  actual  features  as  well  as  on 
changes  in  features.  Effectively,  this  has  been  accomplished  by  adding  temporal  changes  in  features, 
averaged  over  images,  to  the  feature  vector  already  used  in  our  CAD  algorithms.  This  classification 
mechanism  has  been  implemented  as  a  neural  network,  and  trained  using  the  training  subset  of  our 
dataset.  This  is  performed  locally  for  sequences  of  ipsilateral  exams,  and  the  results  of  these 
classifications  are  displayable.  The  feature  set  used  as  input  to  the  classifier  includeds: 

•  Local  density 

•  Local  trend  of  changes  in  density 

•  Local  variance 

•  Local  trend  of  changes  in  variance 

•  Probability  of  malignancy  from  multi- view  CAD 

The  system  has  been  designed  to  allow  additional  features  to  be  easily  incorporated,  once  they  are  defined 
and  implemented. 

An  intense  value  in  one  of  these  images  corresponds  to  a  high  probability  of  the  existence  of  a  malignancy 
at  the  particular  location. 

Task  6:  Interim  testing  and  analysis  (Months  18  -  22) 

We  have  developed  methods  and  software  to  measure  the  contribution  of  each  of  the  temporal  features 
to  the  overall  performance  of  the  classifier  defined  above.  The  neural  network  classifier  is  trained  and 
tested  on  the  training  set  of  cases  using  a  jackknife  procedure,  both  with  and  without  a  feature  of  interest. 
Performance  is  measured  by  ROC  analysis,  with  the  Az  value  being  used  as  a  figure-of  merit.  The 
difference  in  performance,  with  or  without  the  feature,  is  taken  as  a  measure  of  the  importance  of  the 
particular  feature. 

Task  7:  Final  training  and  testing  of  classifier  (Months  24-30) 

Our  method  for  selecting  an  optimal  feature  set  by  using  genetic  algorithms  has  been  adapted  for 
this  project.  But  because  the  set  of  features  implemented  to  date  is  small,  and  we  have  paid  particular 
attention  to  insure  that  they  are  independent,  the  method  is  not  actually  useful  at  this  point.  It  will  become 
more  important  during  the  one-year  extension  of  the  project  as  texture  features  are  added  that  are  not 
expected  to  be  independent. 

The  main  significant  difference  between  the  final  version  (to  date)  and  the  interim  version  of  the 
classification  mechanism  is  that  we  have  now  included  age  as  a  feature.  This  was  necessary  because 
trends  in  other  parameters,  such  as  breast  density,  tend  to  be  age  dependent. 

Task  8:  Final  processing  of  all  cases  (Months  30  -  34) 

Within  the  past  year,  all  cases  in  the  available  dataset  have  been  processed  by  the  finalized 
methods  described  above  and  the  images  have  been  placed  on  the  workstation  developed  for  this  project. 
While  evaluation  in  terms  of  efficacy  was  not  described  in  the  original  Statement  of  Work,  we  are 
currently  soliciting  subjective  opinions  of  radiologists. 
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Task  9:  Data  analysis  and  final  report  (Months  30-36) 

The  project  was  intended  to  be  primarily  of  a  developmental  nature,  and  no  quantitative  evaluation  of 
overall  performance  has  been  performed  to  date.  However,  the  methods  that  have  been  developed  were 
displayed  at  the  Era  of  Hope  Meeting  in  2005.  Other  manuscripts  summarizing  the  work  have  been 
prepared  and  are  in  the  process  of  being  reviewed.  These  documents  will  provide  details  on  the  relative 
contributions  of  each  of  the  local  features  studied  in  this  project 


One-Year  Extension  -  This  project  was  intended  to  be  primarily  of  a  developmental  nature,  and  no 
meaningful  evaluation  of  the  work  has  been  performed  to  date.  However,  prior  to  testing  the  system  we 
believe  it  is  desirable  to  implement  certain  other  features  that  should  be  included  in  the  evaluation. 

Within  the  scope  of  the  project  we  implemented  several  filters,  as  specified  in  the  original  Statement  of 
Work,  that  measure  characteristics  of  mammograms  that  can  be  followed  throughout  a  woman's 
mammographic  history.  While  doing  this,  it  became  apparent  that  there  were  many  other  features  that 
could  turn  out  to  be  important,  but  which  were  not  within  the  aims  of  the  project.  Nevertheless,  some  of 
these  should  be  implemented  before  a  quantitative  evaluation  of  the  system  is  performed.  The  main 
example  of  such  features  would  be  measures  of  image  texture  (e.g.,  fractal  dimension),  which  have  been 
receiving  increasing  attention  for  their  potential  relevance  in  predicting  risk  of  malignancy.  In  any  event, 
the  filters  developed  within  the  project  can  serve  as  templates  to  make  the  task  of  creating  new  filters 
much  easier. 

A  thorough  evaluation  of  the  project,  such  as  by  an  ROC  study,  is  not  feasible  at  this  time  because  of 
limitations  of  our  current  dataset,  and  the  cost  and  effort  required.  However,  it  will  be  possible  to  have  the 
system  evaluated  subjectively  by  radiologists  within  the  coming  year.  Such  an  evaluation  will  provide  the 
kind  of  preliminary  results  that  will  make  it  possible  for  us  to  propose  a  more  formal  ROC  evaluation  in 
the  future. 

We  acquired  the  image  dataset  as  proposed,  though  the  number  of  long  sequences  of  screening 
mammograms,  ending  with  a  positive  biopsy,  was  somewhat  less  than  expected.  It  has  turned  out  that 
finding  biopsy  proven  positive  cases,  having  long  mammographic  histories,  has  been  very  difficult.  This 
is  mainly  due  to  the  fact  that  recommendations  for  mammographic  screening  of  younger  women  have 
historically  been  ambiguous,  and  until  recently  relatively  few  women  routinely  have  had  annual 
mammographic  examinations.  The  cases  we  acquired  were  sufficient  for  the  work  proposed  in  the 
contract,  but  this  left  few  positive  cases  that  are  available  for  testing  the  system.  (It  is  not  meaningful  to 
test  the  system  with  cases  used  in  its  development  and  optimization.)  Discovering  generalizable  trends  by 
using  the  methods  developed  in  this  project  requires  these  longer  exam  histories.  With  this  in  mind, 
database  acquisition  is  an  ongoing  activity  in  our  facility,  and  will  continue  with  funding  from  this  and 
other  projects.  We  are  presently  trying  to  collect  a  sufficient  set  of  these  long  image  sequences  for  us  to 
perform  a  more  rigorous  analysis  of  the  relationship  between  mammographic  trends  and  risk  of 
malignancy.  These  additional  cases  will  also  increase  the  power  of  any  evaluation  study.  This  work  will 
be  completed  during  the  coming  year. 


Key  Research  Accomplishments 

•  Selected,  verified  and  digitized  240  cases,  and  entered  them  into  database. 

•  Modified  workstation  for  this  project. 
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•  Developed  display  software. 

•  Developed  an  innovative  registration  method  for  mammograms. 

•  Developed  Image  analysis  software  including  software  to: 

o  Correct  for  characteristic  curve, 
o  Correct  for  breast  compression, 
o  Detect  Skin  line, 
o  Adjust  for  breast  compression, 
o  Calculate  local  and  global  composition, 
o  Global  composition  from  two  views, 
o  Perform  local  geometric  registration, 
o  Perform  histogram  equalization  between  multiple  images. 

•  Identified  local  and  global  temporal  features. 

•  Used  temporal  features  to  create  parameter  images. 

•  Derived  trend  summary  images  from  the  sequences  of  parameter  images. 

•  Developed  a  classification  mechanism  for  predicting  risk  from  temporal  features. 

•  Provided  a  mechanism  for  ROC  analysis  of  the  performance  of  individual  features. 

•  Provided  a  mechanism  for  optimization  of  feature  sets. 

•  Processed  and  installed  the  dataset  of  cases  on  the  workstation,  for  subjective  evaluation  by 
radiologists. 


Reportable  Outcomes 


Manuscripts 

Chang  YH,  Good  WF,  Wang  XH,  Glenn  S.  Maitz  GS,  Zheng  B,  Hardesty  LA,  Hakim  CM,  Gur  D. 
Integrated  Density  of  a  Lesion:  A  Quantitative,  Mammo graphically-derived,  Invariable  Measure.  Med. 
Phys.  2003;  30(7):  1805-11. 

Wang  XH,  Chapman  BE,  Good  WF.  Evaluation  of  quantitative  measures  of  breast  tissue  density  from 
mammography  with  truth  from  MRI  data.  Proc  SPIE  2003;  5032:82-89. 

Good  WF,  Wang  XH,  Maitz  G.  Feature-based  differences  between  mammograms.  Proc  SPIE  2003; 
5032:919-29. 

Zheng  B,  Leader  JK,  Abrams  G,  Shindel  B,  Catullo  V,  Good  WF,  Gur  D.  Computer-aided  detection 
schemes:  the  effect  of  limiting  the  number  of  cued  regions  in  each  case.  AJR  Am  J  Roentgenol. 

2004  Mar;182(3):579-83. 

Chang  YH,  Good  WF,  Leader  JK,  Wang  XH,  Zheng  B,  Hardesty  LA,  Hakim  CM,  Gur  D. 

Integrated  density  of  a  lesion:  a  quantitative,  mammographically  derived,  invariable  measure.  Med 
Phys.  2003  Jul;30(7):  1805-11. 
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Wang  XH,  Good  WF,  Fuhrman  CR,  Sumkin  JH,  Britton  CA,  Warfel  TE,  Herbert  D,  Gur  D.  Stereo  Display 
for  Chest  CT.  SPIE  Electronic  Imaging  2004;  5291:  In  Press 

Wang  XH,  Good  WF,  Fuhrman  CR,  Sumkin  JH,  Britton  CA,  Warfel  TE,  Gur  D.  Projection  Models  for 
Stereo  Display  of  Chest  CT.  SPIE  Medical  Imaging  2004;  5367:  In  Press 

Wang  XH,  Good  WF,  Fuhrman  CR,  Sumkin  JH,  Britton  CA,  Warfel  TE,  Gur  D.  Stereo  Display  for  Chest 
CT.  RSNA  Dec  2003; 

Good  WF,  Wang  XH,  Maitz  G.  Automated  Analysis  and  Display  of  Temporal  Sequences  of  Mammograms.  Era 
of  Hope  2005;  Philadelphia,  PA. 


Databases 


We  have  established  a  database  of  verified  cases,  where  each  case  consists  of  a  sequence  of  screening 
mammography  exams.  While  we  will  continue  to  accrue  cases  to  this  database,  it  is  currently  in  a  form 
that  can  be  used  for  analysis. 


Proposals  applied  for 
US  Army  MRMC  2003 

US  Army  MRMC  2003 
US  Army  MRMC  2003 

Komen  2002 
NIH  2004 
NIH  2004 


Assessing  the  Interdependence  of  Tumor  Risk,  Conspicuity  and  Breast 
Tissue  Characteristics  Derived  from  Mammography 

Computerized  Analysis  and  Display  of  Contralateral  Breast  Asymmetry 

Improving  Breast  Cancer  Assessment  Through  More  Accurate 
Measurement  of  Mass  Size  and  Growth  Rate 

Integrated  Density  -  A  Quantitative  Measure  of  Breast  Lesion 

Improving  Early  Detection  of  Breast  Cancer 

Integrated  Density  -  A  Quantitative  Measure  of  Breast  Lesion 


Conclusions 

Progress  on  this  project  closely  followed  the  schedule  and  methods  outlined  in  the  original 
proposal.  All  tasks  described  in  the  statement  of  work  have  been  undertaken  and  completed  successfully. 
The  tasks  that  were  completed  during  the  past  year  include: 

Task  4\  Image  analysis  software  development  (Months  6  -  30). 

k.  Detection  and  classification  of  trends. 

Task  7:  Final  training  and  testing  of  classifier  (Months  24-30). 

Task  8:  Final  processing  of  all  cases  (Months  30  -  34). 

Task  9:  Data  analysis  and  final  report  (Months  30-36). 
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Throughout  the  project  certain  technical  issues  arose  and  these  are  being  addressed  in  the  one- 
year  extension  that  has  been  granted.  We  expect  to  be  able  to  complete  this  extra  work  during  the  coming 
year. 

We  remain  optimistic  about  the  prospects  for  the  methods  developed  in  this  project  to  have  a 
positive  impact  on  Radiology.  The  workstation  we  have  developed  will  make  it  possible  for  radiologists 
to  efficiently  view  a  woman's  entire  mammographic  history,  as  well  as  view  summarized  presentations  of 
that  history.  We  expect  that  the  viewing  of  this  additional  data  will  increase  radiologists'  understanding  of 
the  kinds  of  changes  that  are  occurring  in  breasts,  and  the  rapidity  with  which  these  changes  are 
occurring.  Ultimate,  this  will  likely  result  in  increased  efficacy  for  screening  mammography.  During  the 
coming  year,  the  work  will  be  published  and  presented  at  meetings,  in  an  effort  to  introduce  some  of  the 
concepts  to  a  wider  range  of  potential  researchers. 
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Appendices 


1.  Good  WF,  Wang  XH,  Maitz  G.  Automated  Analysis  and  Display  of  Temporal  Sequences  of 
Mammograms.  Era  of  Hope  2005;  Philadelphia,  PA. 

In  reading  screening  mammography,  radiologists  customarily  only  compare  current  exams  to  most  recent 
prior  exams.  Based  on  the  hypothesis  that  longer  sequences  of  screening  exams  contain  information  that 
is  not  currently  being  exploited  in  the  diagnostic  process,  we  have  developed  a  display  system  for 
efficiently  presenting  information  about  changing  tissue  patterns,  derived  from  longer  mammographic 
histories. 

The  system  provides  for:  1)  Deforming  mammograms  so  that  all  images  geometrically  match  the  form  of 
the  corresponding  view  in  the  most  recent  exam;  2)  Correcting  for  differences  in  exposure  between 
exams;  3)  Measuring  local  trends  in  various  parameters  over  time  and  representing  these  trends  in 
summary  images;  and,  4)  Displaying  the  normalized  mammographic  images  in  a  cine  mode,  along  with 
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summary  images.  The  parameters  that  have  been  implemented  to  date  are  local  breast  density  and  local 
variance  of  pixel  values,  which  provide  a  general  prototype  mechanism  for  other  parameters  to  be  added 
in  the  future. 

The  fully  automatic  geometric  transform  can  map  any  mammographic  image  onto  a  reference  image 
while  guaranteeing  registration  of  specific  features  and  maintaining  grayscale  equivalence.  A  polar 
coordinate  system  is  established  with  its  origin  at  the  intersection  of  the  nipple  axis  with  the  pectoral 
muscle.  A  tissue  pixel  is  identified  by  coordinates  consisting  of:  1)  a  relative  distance  between  the  origin 
and  skin  line;  and  2)  its  angle  relative  to  the  nipple  axis.  Pixels  are  mapped  onto  a  position  in  a  reference 
image  determined  by  these  relative  coordinates.  This  transformation  guarantees  that  pectoral  lines,  nipples 
and  skin  lines  are  all  placed  in  complete  registration.  Images  are  then  adjusted  by  converting  film  density 
to  log-exposure  and  dividing  by  the  Jacobian  of  the  geometric  transformation  to  correct  for  local 
expansion  factors.  Relative  exposures  are  equalized  with  respect  to  the  reference  image  by  applying  linear 
regression  to  adjust  corresponding  pixel  value  pairs. 

After  this  normalization  process,  each  image  in  a  sequence  is  processed  by  two  feature-detection  filters 
that  measure  local  tissue  density  and  local  variance  at  each  point.  To  measure  local  density,  the  fraction 
of  fibroglandular  tissue  projected  onto  each  pixel  is  determined  and  represented  in  an  image,  which  is 
then  convolved  with  a  Gaussian  kernel.  At  each  location,  trends  over  time  in  parameters  are  calculated 
and  represented  in  summary  images.  For  each  parameter,  the  slope  of  its  regression  line,  and  its  variance 
with  respect  to  its  regression  line,  are  mapped  onto  a  2D  color  scale. 

For  subjective  evaluation  by  radiologists,  the  system  has  been  loaded  with  240  verified  cases  for  which 
we  have  at  least  three  screening  exams,  however  an  overall  objective  evaluation  of  the  system  is  not 
within  the  scope  of  the  current  project. 


